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Abstract. Rare properties remain a challenge for statistical model check¬ 
ing (SMC) due to the quadratic scaling of variance with rarity. We ad¬ 
dress this with a variance reduction framework based on lightweight im¬ 
portance splitting observers. These expose the model-property automa¬ 
ton to allow the construction of score functions for high performance 
algorithms. 

The confidence intervals defined for importance splitting make it appeal¬ 
ing for SMC, but optimising its performance in the standard way makes 
distribution inefficient. We show how it is possible to achieve equiva¬ 
lently good results in less time by distributing simpler algorithms. We 
first explore the challenges posed by importance splitting and present an 
algorithm optimised for distribution. We then define a specific bounded 
time logic that is compiled into memory-efficient observers to monitor 
executions. Finally, we demonstrate our framework on a number of chal¬ 
lenging case studies. 


1 Introduction 

The ‘state explosion problem’ [B] associated with probabilistic model checking 
has been well addressed by statistical model checking (SMC) [28]. SMC includes a 
number of approximative techniques based on Monte Carlo sampling }24j , which 
only generate states on the fly during simulation. The performance of SMC is 
typically independent of the size of the state space [25] . while simulation cost 
may be divided linearly on parallel computation architectures. Rare properties 
pose a problem because the standard and relative errors scale quadratically with 
rarity nn. For example, 4000 simulations would be sufficient to estimate a 
probability of 0.1 ±10% with 95% confidence, whereas 4x 10 13 simulations would 
be necessary to estimate a probability of 10~ 6 ± 10% with the same confidence. 
Since quantifying rare properties is often important to certify the reliability 
of complex critical systems, we seek to enhance SMC with variance reduction 
techniques, such as importance sampling and importance splitting I2llldl26| . 
while still taking advantage of the easy distribution that SMC typically affords. 

Importance sampling weights the executable model of a system so that the 
rare property occurs more frequently in simulations. The proportion of simula¬ 
tions that satisfy the property using the weighted model overestimates the true 
probability, but the estimate may be exactly compensated by the weights. It is 
generally not feasible to implement a perfectly weighted executable model for 






importance sampling because (i) the perfect model may not actually exist as 
a re-parametrisation of the original model and ( ii) a perfect re-parametrisation 
typically requires an iteration over all the transitions, defeating the benefits of 
sampling. Practical approaches tend to use a low dimensional vector of param¬ 
eters to weight the model M. Given such a parametrisation, importance 
sampling can be implemented with minimal memory and may be distributed 
efficiently on parallel computational architectures. The principal limitation of 
importance sampling is that without a guarantee that the simulation model is 
perfect, it is difficult to formally bound the error of estimates. In contrast, useful 
confidence intervals have been defined for importance splitting |514j . 

Importance splitting divides a rare property into a set of less rare sub¬ 
properties that correspond to an increasing sequence of disjoint levels: the initial 
state corresponds to the lowest level, while states that satisfy the rare property 
corresponds to the final level. Importance splitting algorithms use a series of 
easy simulation experiments to estimate the conditional probabilities of going 
from one level to the next. Since relatively few simulations fail to satisfy the 
sub-properties, the overall simulation budget may be reduced. Each experiment 
comprises simulations initialised with the terminal states of previous simulations 
that reached the current level. The overall probability is the product of the es¬ 
timates, with the best performance (lowest variance) achieved with many levels 
of equal conditional probability. 

Importance splitting poses several challenges for optimisation and distribu¬ 
tion. In the context of SMC, importance splitting algorithms repeatedly initialise 
simulations with states of the model-property product automaton. For arbitrary 
properties this may have size proportional to the length of a simulation trace. 
At the same time, increasing the number of levels to maximise performance re¬ 
duces the number of simulation steps in each simulation experiment. The cost 
of sending the model-property state across slow communication channels may 
be significantly greater than the cost of short simulations. In addition, to spec¬ 
ify levels with equal conditional probabilities it is necessary to define a ‘score 
function’ that maps the states of the product automaton to a value. This cannot 
easily be automated, so a syntactic description of the property automaton must 
be accessible for the user to construct a score function manually. 


To address the above challenges we present an importance splitting frame¬ 
work for SMC, specifically considering the problems of distribution. We first 
discuss the problems of distributing importance splitting algorithms and present 
a fixed level algorithm optimised for distribution. We then define an expressive 
bounded time temporal logic and describe the system of efficient lightweight 
observers that implement it. These make the product automaton (*) accessi¬ 
ble to the user, (ii) efficient to construct, (Hi) efficient to distribute and (iv) 
efficient to execute. Finally, we demonstrate the performance and flexibility of 
our framework on a number of case studies that are intractable to numerical 
methods. 




Related Work 


There have been many ad hoc implementations of importance splitting based on 
the original ideas of mm . The algorithm of m is a relatively recent example 
that is often cited. The work of m is novel because the authors define efficient 
adaptive importance splitting algorithms that also include confidence intervals. 
To our knowledge, [T8] is the first work to explicitly link importance splitting 
to arbitrary logical properties, while the present work is the first to describe a 
practical importance splitting framework for SMC. The present work is thus the 
first to consider the problems of distributing importance splitting for SMC. 

SMC tools construct an automaton (a monitor) to accept traces that satisfy 
a temporal logic formula, typically based on a time bounded variant of temporal 
logic. The proportion of independent simulations of a stochastic model that 
satisfy the property is then used to estimate the probability of the property 
or to test hypotheses about the probability. There is considerable intersection 
between runtime verification (RV) and SMC, with few concepts unique to either. 
In particular, there have been many works that construct RV monitors from 
temporal logic (e.g., Il0ll2ll4l9l2h . Such monitors typically comprise tableau- 
based automata m whose states represent the combinations of subformulas of 
the overall property. While some have considered timed properties (e.g. ®), the 
focus is predominantly unbounded LTL properties interpreted on finite paths [8] . 
In contrast, SMC typically checks formulas with explicit time bounds (see, e.g., 
©), which are inherently defined on finite traces. To avoid the combinatorial 
explosion of subformulas caused by including time in this way, the monitors 
used by mm and other tools are compact “programs” that generate the states 
of an automaton on the fly and do not store them. We adapt this “lightweight” 
approach to allow importance splitting for SMC to be efficiently distributed on 
high performance parallel computational architectures. 


2 Technical Background 

Our SMC tools |17l3j implement a bounded linear temporal logic having the 
following typical syntactic form: 

0 = X fe </> | F k cj> | G k (j) | cj)\5 k (t) | -■</> \ <f>\/(t>\(t>/\(t>\(t>=><t>\a (1) 

This syntax allows arbitrary combinations and nesting of temporal and atomic 
properties (i.e., those which may be evaluated in a single state and denoted by 
a). The time bound k may denote discrete steps or continuous time, but in this 
work we consider only discrete time semantics. 

Given a finite trace w, comprising sequence of states W 0 W 1 W 2 ■ • •, de¬ 
notes the suffix • The semantics of the satisfaction relation \= is 









constructed inductively as follows 
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Other elements of the relation are constructed using the equivalences false = 
-i true , (f>A(f> = — 1(— V-■</>), F k <f> = true\J k (j), G k <f> = -<(trueXJ k -*(/)). Hence, given 
a property <p, with syntax according to m, to |= ip is evaluated by |= p. 


Importance Splitting and Score Functions 

The neutron shield model of |20I21J is illustrative of how importance splitting 
works. The distance travelled by a neutron in the shield defines a monotonic 
sequence of levels 0 = sq < s\ < S 2 < • • • < s m = shield thickness, such that 
reaching a given level implies having reached all the lower levels. While the 
overall probability 7 of passing through the shield is small, the probability of 
passing from one level to another can be made arbitrarily close to 1 by reducing 
the distance between levels. Denoting the abstract level of a neutron as s, the 
probability of a neutron reaching level Sj can be expressed as P(s > s,) = P(s > 
Si | s > Si_i)P(s > s,_i). Defining 7 = P(s > s m ) and P(s > sq) = 1, 


7 = n P(s > Si I s > Si- 1). (3) 

i =1 

Each term of © is necessarily greater than or equal to 7 , making their estimation 
easier. By writing 7 * = P(s > Si | s > Sj_ 1 ) and denoting the estimates of 7 and 
7 i as respectively 7 and 74 , [TS] defines the unbiased confidence interval 
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Confidence is specified via z a , the 1 — a/2 quantile of the standard normal 
distribution, while n is the per-level simulation budget. We infer from © that 
for a given 7 the confidence is maximised by making both the number of levels 
m and the simulation budget large, with all 74 equal. 

The concept of levels can be generalised to arbitrary systems and properties in 
the context of SMC, treating s and ,s 7 ; in © as values of a score function over the 
model-property product automaton. Intuitively, a score function discriminates 
good paths from bad, assigning higher scores to paths that more nearly satisfy 







the overall property. Since the choice of levels is crucial to the effectiveness of 
importance splitting, various ways to construct score functions from a temporal 
logic property are proposed in j!8j . Formally, given a set of finite trace prefixes 
uj £ 17, an ideal score function S : f2 —> R has the characteristics S(u>) > 
S(uj') <£=> P(|= p | uj) > P(|= p | uj'), where P()= p | uj) is the probability of 
eventually satisfying p given prefix uj. Intuitively, uj has a higher score than uj' 
iff there is more chance of satisfying p by continuing uj than by continuing uj' . 
The minimum requirement of a score function is S(u >) > s v <==>• uj |= p, where 
Stp is an arbitrary value denoting that tp is satisfied. Any trace that satisfies p 
must have a score of at least s v and any trace that does not satisfy p must have 
a score less than s v . In what follows we assume that © refers to scores. 

3 Distributing Importance Splitting 

Simple Monte Carlo SMC may be efficiently distributed because once initialised, 
simulations are executed independently and the result is communicated at the 
end with just a single bit of information (i.e., whether the property was satisfied 
or not). By contrast, the simulations of importance splitting are dependent be¬ 
cause scores generated during the course of each simulation must be processed 
centrally. The amount of central processing can be minimised by reducing the 
number of levels, but this generally reduces the overall performance. 

Alternatively, entire instances of the importance splitting algorithm may be 
distributed and their estimates averaged, with each instance using a proportion¬ 
ally reduced simulation budget. We use this approach to generate some of the 
results in Section [(3 but note that if the budget is reduced too far, the algorithm 
will fail to pass from one level to the next and no valid estimate will be produced. 

Distribution of importance splitting is thus possible, but its efficiency is de¬ 
pendent on the particular problem. In this work we therefore provide the frame¬ 
work to explore different approaches. In Section 13.11 we first describe the con¬ 
cept of an adaptive importance splitting algorithm and then explain why this 
otherwise optimised technique is unsuitable for distribution. In Section 13.21 we 
motivate the use of a fixed level algorithm for “lightweight” distribution and 
provide a suitable algorithm. The results we present in Section [G] demonstrate 
that this simpler approach can be highly effective. 


3.1 The Adaptive Algorithm 

The basic notion of importance splitting described in Section [2] can be directly 
implemented in a so-called fixed level algorithm, i.e., an algorithm in which the 
levels are pre-defined by the user. With no a priori information, such levels will 
typically be chosen to subdivide the maximum score equally. In general, how¬ 
ever, this will not equally divide the conditional probabilities of the levels, as 
required by © to maximise performance. In the worst case, one or more of the 
conditional probabilities will be too low for the algorithm to pass between levels. 


Finding good or even reasonable levels by trial and error may be computation¬ 
ally expensive and has prompted the development of adaptive algorithms that 
discover optimal levels on the fly |5ll8ll9j . Instead of pre-defining levels, the 
user specifies the proportion of simulations to retain after each iteration. This 
proportion generally defines all but the final conditional probability in (pi) . 

The adaptive importance splitting algorithm first performs a number of sim¬ 
ulations until the overall property is decided, storing the resulting traces of the 
model-property automaton. Each trace induces a sequence of scores and a corre¬ 
sponding maximum score. The algorithm finds a level that is less than or equal 
to the the maximum score of the desired proportion of simulations to retain. The 
simulations whose maximum score is below this current level are discarded. New 
simulations to replace the discarded ones are initialised with states correspond¬ 
ing to the current level, chosen at random from the retained simulations. The 
new simulations are continued until the overall property is decided and the pro¬ 
cedure is repeated until a sufficient proportion of simulations satisfy the overall 
property. 

The principal advantage of the adaptive algorithm is that by simply rejecting 
the minimum number of simulations at each level it is possible to maximise 
confidence for a given score function. The principal disadvantage is that it stores 
simulation traces, severely limiting the size of model and simulation budget. 
The use of lightweight computational threads is effectively prohibited. Moreover, 
minimising the number of rejected simulations reduces the number of simulations 
performed between levels, thus reducing the possibility to perform computations 
in parallel. Minimising the rejected simulations also maximises the number of 
levels, which in turn minimises the number of simulation steps between each 
level. This further limits the feasibility of dividing the algorithm, since sending 
a model-property state over a slow communication channel may be orders of 
magnitude more costly than performing a short simulation locally. 


3.2 A Fixed Level Algorithm for Distribution 

In contrast to the adaptive algorithm, the fixed level importance splitting al¬ 
gorithm does not need to store traces, making it lightweight and suitable for 
distribution. Scores are calculated on the fly and only the states that achieve 
the desired level are retained for further consideration. While the choice of levels 
remains a problem, an effective strategy is to first use the adaptive algorithm 
with a relatively high rejection rate to find good fixed levels. An estimate with 
high confidence can then be generated efficiently by distributing the fixed level 
algorithm. 

Algorithm |T| is our fixed level importance splitting algorithm optimised for 
distribution. We use the terms server and client to refer to the root and leaf 
nodes of a network of computational devices or to mean independent computa¬ 
tional threads on the same machine. In essence, the server manages the job and 
the clients perform the simulations. The server initially sends compact represen¬ 
tations of the model and property to each client. Thereafter, only the state of 




the product automaton is communicated. In general, each client returns termi¬ 
nal states of simulations that reached the current level and the server distributes 
these as initial states for the next round of simulations. Algorithm [1] optimises 
this. The server requests and distributes only the number of states necessary to 
restart the simulations that failed to reach the current level, while maintaining 
the randomness of the selection. Despite this optimisation, however, the perfor¬ 
mance of this and other importance splitting algorithms will be confounded by 
the combination of large state size and properties having short time bounds. 
Under such circumstances it may be preferable to distribute entire instances of 
the algorithm, as described above. 

The memory requirements of Algorithm [l] are minimal. Each client need 
only store the state of n simulations. As such, it is conceivable to distribute 
simulations on lightweight computational threads, such as those provided by 
GPGPU (general purpose computing on graphics processing units). 


Algorithm 1: Distributed Fixed Level Importance Splitting 
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input: si < S 2 < • • • < s m is a sequence of scores, with s m = s v the score 
necessary to satisfy property ip 
7 <r- 1 is the initial estimate of 7 = P(cj \= p) 

server sends compact description of model and observer to k clients 
each client initialises n simulations 
for S i — Sl, . . . , Sm do 

each client continues its n simulations from their current state 
simulations halt as soon as their scores reach s 

V clients, client i sends server the number of traces n; that reached s 
server calculates 7 <— 7 'n'/kn, where ri = ^ rii 
for j 1, ..., kn — n! do 

server chooses client i at random, with probability m/n' 

client i sends server a state chosen uniformly at random from those that 

reached s 

server sends state to client corresponding to failed simulation j, as 
initial state of new simulation to replace simulation j 


output: 7 


4 Linear Temporal Logic for Importance Splitting 

High performance SMC tools, such as nzm], avoid the complexity of standard 
model checking by compiling the property to a program of size proportional 
to the formula and memory proportional to the maximum sum of nested time 
bounds. This program implicitly encodes the model checking automaton, but is 
exponentially smaller. For example, the property X fc </> can be implemented as a 
loop that generates k simulation steps before returning the truth of <p in the last 
state; the property ipXJ k (f> can be implemented as a loop that generates up to k 









simulation steps while ip is true and <p is not true, returning the value of <p in the 
last state otherwise. If ip and cp are atomic, the programs require just O(logfc) 
bits of memory to hold a loop counter. 

In contrast, the nested property F kl (ip V G k 2 (p) has an 0(k2) memory re¬ 
quirement. If ip is not true on step i < k 1 it may be necessary to simulate up 
to step i + k2 to decide subformula G k 2 cp. If ip V G k2 (p turns out to be false 
on step *, it will then be necessary to consider the truth of ip on step i + 1 , 
noting that the last simulated step could be i + k2. To evaluate this formula it 
is effectively necessary to remember the truth of ip on 0(k2) simulation steps. 
Similar requirements can arise when the until operator (U) is a subformula of a 
temporal operator. In all such cases, the sequence of stored truth values become 
part of the state of the property automaton. 

SMC using importance splitting requires that simulations are repeatedly and 
frequently initialised with the state of the model-property product automaton. 
If the size of this state is proportional to the time bounds of temporal opera¬ 
tors, initialisation may have comparable complexity to simulation. This becomes 
especially problematic if the state is to be transmitted across relatively slow 
communication channels for the purposes of distribution. We therefore define a 
subset of 0, the size of whose automata is not dependent on the bounds of 
temporal operators: 

cp =X k cp | ip\J k ip \-«p\(pV<p\(pA<p\(p^(p\'ip 

ip =X. k ip | F k ip | G k ip ] a ' 

The semantics of ([5]) is the same as m, but ([5]) restricts how temporal op¬ 
erators may be combined. In particular, U may not be the subformula of a 
temporal operator other than X and temporal operators that are subformulas 
of other temporal operators may not be combined with Boolean connectives. 
Temporal operators containing other temporal operators as subformulas may, 
however, be combined. This logic expresses many useful properties, including 
nested bounded temporal properties that are not implemented in Prism. 

5 Lightweight Observers for Importance Splitting 

To facilitate the construction of score functions we implement the logic given by 
(0 as a set of nested observers. Each observer corresponds to either a temporal 
operator, a Boolean operator acting on temporal operators, or as a predicate 
describing an atomic property. In our implementation observers are written in 
a syntax based on the commonly used reactive modules language [I], using the 
notion of ‘guarded commands’ [7] with sequential semantics. 

An observer comprises a set of guarded commands, any number of which may 
be enabled and executed on a given simulation step. Updates are performed in 
syntactic order after all guards have been evaluated, hence the update of one 
command does not affect the guards of commands in the same observer. In 
general, the output of one observer is the input to another and observers are 
therefore executed in reverse order of their nesting. 


Observers evaluate states as they are generated by the simulation. Since it 
may not be possible to decide a property before seeing a certain number of states, 
observers implement a three valued logic. In Figs. [TJ [5] and [3] we use the symbols 
?, T and _L to denote the three values undecided, true and false, respectively. 
The state of an observer changes only when at least on of its inputs is decided. 
An observer may reach a deadlock state (no commands enabled) once its output 
is decided and cannot be changed by further input. A simulation terminates 
when the output of the root observer is decided, i.e., the property is decided. 
Simulations may also be paused by the importance splitting algorithm if the 
score reaches a desired level. 

Observers implementing the same temporal operator behave differently ac¬ 
cording to their level of nesting within a formula. We therefore distinguish outer 
and inner temporal observers. The temporal operators closest to the root of 
any branch of the syntax tree induced by a formula are implemented by outer 
observers. Their output proceeds from undecided to either true or false and then 
does not change. Inner observers encode temporal operators that are the subfor¬ 
mulas of other temporal operators. Their output proceeds from undecided to a 
possibly alternating sequence of true, false and undecided values because their 
enclosing operator(s) cause them to evaluate a moving widow of states. The in¬ 
ner and outer variants of X, F and G are closely related—outer observers are 
essentially simplified inner observers. When U is a subformula of X, however, 
the X is implemented as a delay within the U observer. 

In what follows we describe the important aspects of the various observers 
that implement The accompanying figures include a diagrammatic repre¬ 
sentation of how the observers work and a set of commands written in the form 
predicate : update. Each observer has Boolean output variables o and d to indi¬ 
cate respectively the result and whether the property has been decided (observers 
for atomic formulas omit d). Observers for temporal operators take discrete time 
bound k as a parameter and use a counter variable w (U uses counter variables 
w' and w"). Inner temporal operators make use of an additional counter, t (U 
uses t' and t"). The inputs of observers are Boolean variables o' and o", with 
corresponding decidedness d' and d". 


Connective Observers These observers implement Boolean connectives at 
syntactic level <f> in (0 and take advantage of the equivalences false A ? = false, 
true V ? = true, false => ? = false and ? => true = true, for any truth value 
of ?. Figure fTal describes the observer for conjunction and Fig. [Tb] describes the 
observer for implication. The observer for disjunction may be derived from that 
of conjunction by negating all instances of o' and o", and by exchanging o 4— true 
and o <— false. Negation is implemented by inverting the truth assignment of 
the observer to which it applies, i.e., by exchanging o true and o <r- false. The 
connectives may be combined with themselves and outer temporal operators. 
Boolean connectives that apply only to atomic properties (i.e., syntactic level a) 
are implemented directly in formulas within observers for atomic properties. 
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1. —id A (~i d' V ~id 77 ) A — 1 (—>o 7 A d! V —io 77 A d 77 ) 

2. -id A d! A o' A d 77 A o" : d 4— true, o 4— true 

3. -id A (-ic/ Ad'v -io” A d") : d 4— true, o 4— false 


(a) 


o' A o" 



1. —id A (—id 7 A —i(d 77 A o 77 ) V d 7 A o' A —id 77 ) 

2. -id A (-io 7 A d 7 V o 77 A d 77 ) : d 4— tree, o 4- £r«e 

3. -id A d 7 A o 7 A d 77 A -io 77 : d 4— £r«e, o 4— false 

(b) o-^o' => o" 


Fig. 1: Connective observers. Initially d = false. 


Inner Temporal Observers These observers act on a moving window of states 
created by an enclosing temporal operator. The output may pass from one de¬ 
cided value to the other and also become undecided. 

Figure l2al describes the observer for X fc . Command 1 counts decided input 
states until bound k is reached. Thereafter command 2 sets the output decided 
and equal to the value of the input. 

Figure [2b] describes the observer for F fe . While decided inputs are not true, 
command 1 increments w from 0 to k. If at any time the input is true, command 
2 sets the output to true and the “true-counter” t is set to w. Command 5 
decrements t on subsequent false inputs. The output remains true while t > 0. 
If w reaches k while t = 0, command 3 sets the output to false. 

The observer for G fc may be derived from that of F fe by negating all instances 
of o' and -i o', and by exchanging o 4— true and o 4— false. 

Outer Temporal observers The outer observers for X. k and F fc are not illus¬ 
trated but may be derived from their respective inner observers given in Fig. [2] 
For X fc , command 3 is removed and the guard of command 2 is strengthened 
with -i d. For F fc , commands 4, 5 and 6, together with all references to counter 
t, are removed, while the guards of commands 2 and 3 are strengthened by -> d. 
The outer observer for G k can be derived from that of F fc in the same way as 
described for inner temporal observers. 

Figure [3] describes the observer for properties of the form X fex (ip\J k (p) and 
ip\J k <p. Since ip and <j> may be temporal formulas that are satisfied on different 
simulation steps in arbitrary order, the observer employs variables w' and w" 
to respectively count the sequences of —>(p and 0 (commands 3 and 5). Variable 
t' then records the position of the first cj> (command 4), while t" records the 
position of the last ip (command 5). Using t' and t", commands 7 and 8 are able 
to determine if the property is satisfied or falsified, respectively. 
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1. -*d A d' A w < k : w <r- w + 1 

2. d' A w = k : d «— true, o 4— o' 

3. d A -id' : d <— false 


(a) o e- XV 
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1. -id Ad' A -io' Aw<k:w<—w + 1 

2. / A o' : o f- true, d «— true, t <— w 

3. d' A -io / At = 0Aw = k:d<— true, o «— false 

4. d Ad' A -io' At = 0Aw<k:d<^~ false 

5. d Ad' A -io' At > 0 : t <— t — 1 

6. d A -id' : d <— false 

(b) o F V 


Fig. 2: Observers for inner temporal operators. Initially w = t = 0, d = false. 


Properties of the form X fcx (^U fc 0) are implemented by simply initialising 
variables w' and re" to — fcx, forcing the observer to ignore the first fcx decided 
values of ip and <f>. If the property is not of this form, w' and w" are initialised 
to 0 and the automaton may be simplified by removing commands 1 and 2 and 
all instances of expressions w 1 > 0 and w" > 0. 


1 , 2 , 3 , 4 , 5,6 



1. d! A w' < 0 : w' w' + 1 

2. d" A w" < 0 : w" <r- w" + 1 

3. ->d Ad' A -io' Ati)'>0As)'<l:«)Vtii' + l 

4. -id A d' A o' A w' > 0 A w < k : t! <— w ', u/ <— fc + 1 

5. -id A d" A o" A w” > 0 A w” < k : w" <— w" + 1, t" ui” 

6. -id A d" A -io" A w" > 0 A w" < k : to" <— k 

7. -id A t' > 0 A t" > t' — 1 : d <— true, o true 

8. -id A (t' < 0 A u/ = fc + 1 V w" = k A (t" < t' — 1 

Vt r < 0 A f" < w/ — 1)) : d <— true, o false 


Fig. 3: Observer for o X fcx (o"U fc o'). Initially t" = 0, t' = — 1, d = dx = false 
and w/ = to" = —A;x (see text). 


6 Case Studies 

We have implemented our importance splitting framework in Plasma [ 3 ] and 
demonstrate its use on three case studies whose state space is intractable to nu¬ 
merical model checking. The following results do not seek to promote a particular 
methodology (adaptive or fixed level algorithm, distributed or single machine), 


but serve to illustrate the flexibility of our platform. The software, models and 
observers can be downloaded from our websitcfl The leader election and dining 
philosophers models are also illustrated on the Prism case studies websitc@. 

For each model we performed a number of experiments to compare the perfor¬ 
mance of the fixed and adaptive importance splitting algorithms with and with¬ 
out distribution, using different budgets and levels. Our results are illustrated 
in the form of empirical cumulative probability distributions of 100 estimates, 
noting that a perfect (zero variance) estimator distribution would be a single 
step. The results are also summarised in Tabic [I] The probabilities we estimate 
are all close to 10 -6 and are marked on the figures with a vertical line. Since we 
are not able to use numerical techniques to calculate the true probabilities, we 
use the average of 200 low variance estimates as our best overall estimate. 

As a reference, we applied the adaptive algorithm to each model using a 
single computational thread. We chose parameters to maximise the number of 
levels and thus minimise the variance for a given score function and budget. 
The resulting distributions, sampled at every tenth percentile, are plotted with 
circular markers in the figures. Over these points we superimpose the results of 
applying a single instance of the fixed level algorithm with just a few levels. We 
also superimpose the average estimates of five parallel threads running the fixed 
level algorithm, using the same levels. 

The figures confirm our expectation that the fixed level algorithm with few 
levels is outperformed by the adaptive algorithm. The figures also demonstrate 
that the average of parallel instances of the fixed level algorithm are very close 
to the performance of the adaptive algorithm. The timings given in Table[l]show 
that the distributed approach achieves these results in less time. For comparison 
we also include the estimated time of using a simple Monte Carlo (MC) estimator 
to achieve the same standard deviation. Importance splitting gives more than 
three orders of magnitude improvement in all cases. All results were generated 
using an Intel Core i7-3740 CPU with 4 cores running at 2.7 GHz. 

In the remainder of this section we briefly describe our models and their 
associated properties and score functions. 

Leader Election Our leader election case study is based on the Prism model 
of the synchronous leader election protocol of m • With N = 20 processes and 
K = 6 probabilistic choices the model has approximately 1.2 x 10 18 states. We 
consider the probability of the property G 420 -i elected, where elected denotes the 
state where a leader has been elected. Our chosen score function uses the time 
bound of the G operator to give nominal scores between 0 and 420. The model 
constrains these to only 20 actual levels, but with evenly distributed probability. 
For the fixed level algorithm we use scores of 70,140, 210, 280,350 and 420. 

Dining Philosophers Our dining philosophers case study extends the Prism 
model of the fair probabilistic protocol of [23]. With 150 philosophers our model 

1 projects.inria.fr / plasma-lab / importance-splitting 

2 www.prismmodelchecker.org/casestudies 
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Fig. 4: Leader election. 


Fig. 5: Dining philosophers. 


contains approximately 2.3 x 10 144 states. We consider the probability of the 
property F 30 Phil eats , where Phil is the name of an arbitrary philosopher. The 
adaptive algorithm uses the heuristic score function described in [19] . which 
includes the five logical levels used by the fixed level algorithm. The heuristic 
favours short paths, based on the assumption that as time runs out the property 
is less likely to be satisfied. 



Adaptive 

Single 

Parallel 

Std. dev. 4.8 X 10 -8 

1.3 X icr 7 5.2 x icr 8 

Levels 20 

6 

6 

Budget 1000 

1000 

5 X 1000 

Time (MC) 7.3s (30h) 

2.5s (4.4h) 5.8s (5.Oh) 

Std. dev. 4.2 X 10 -7 

7.7 X 10 -7 2.8 X 10~ 7 

Levels 109 

5 

5 

Budget 1000 

1000 

5 X 1000 

Time (MC) 5.4s (2.3h) 

1.7s (41m) 3.7s (1.4h) 

Std. dev. 2.1 X 10 -7 

5.0 X 10~ 7 2.3 X 10~ 7 

Levels 3942 

4 

4 

Budget 500 

500 

5 X 500 

Time (MC) 15s (7.5h) 

2.8s (1.2h) 4.8s (1.9h) 


Fig. 6: Dependent counters. 


Table 1: Summary of results. 


Dependent Counters Our dependent counters case study comprises ten coun¬ 
ters, initially set to zero, that with some probability dependent on the values of 
the other counters are either incremented or reset to zero. This can be viewed as 
modelling an abstract computational process, a set of reservoirs of finite capac¬ 
ity, or as the failure and repair of ten different types of components in a system, 
etc. With a maximum count of 10, the model has approximately 2.6 x 10 10 states. 

We consider the probability of the property X 1 (-im*tU 1000 complete), where 
init and complete denote the initial state and the state where all counters have 











reached their maximum value. Our score function ranges over values between 0 
and 99, but the probabilities are not evenly distributed. With a budget of 500, 
uniformly distributed fixed scores fail to produce traces that satisfy the property 
until the difference between the last two levels is about 5. Note that our budget 
is limited to only 500 simulations due to the length of the traces that must be 
stored by the adaptive algorithm. We maintain this budget for the fixed level 
algorithm to simplify comparison. After a small amount of trial and error, we 
adopted fixed scores of 80,90,95 and 99. 

7 Challenges and Prospects 

Our results demonstrate the effectiveness and flexibility of our framework with 
discrete time properties applied to standard case studies. Future challenges in¬ 
clude industrial scale examples and the implementation of continuous time prop¬ 
erties. We also intend to provide proofs of the correctness of our observers and 
of our logic’s memory requirements. 

Although the manual construction of score functions adds to the overall cost 
of using importance splitting, we believe that distribution relaxes the need for 
these to be highly optimised. We also expect that it will be possible to construct 
good score functions automatically using statistical learning techniques. 
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